

<!DOCTYPE html>
<html class="writer-html5" lang="zh" >
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>下载和处理文件和图像 &mdash; Scrapy 2.3.0 文档</title>
  

  
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.custom.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.bundle.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-shadow.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-punk.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-noir.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-light.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-borderless.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/micromodal.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/sphinx_rtd_theme.css" type="text/css" />

  
  
  
  

  
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/language_data.js"></script>
        <script src="../_static/js/hoverxref.js"></script>
        <script src="../_static/js/tooltipster.bundle.min.js"></script>
        <script src="../_static/js/micromodal.min.js"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    
    <link rel="index" title="索引" href="../genindex.html" />
    <link rel="search" title="搜索" href="../search.html" />
    <link rel="next" title="部署蜘蛛" href="deploy.html" />
    <link rel="prev" title="调试内存泄漏" href="leaks.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../index.html" class="icon icon-home" alt="Documentation Home"> Scrapy
          

          
          </a>

          
            
            
              <div class="version">
                2.3
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <p class="caption"><span class="caption-text">第一步</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../intro/overview.html">Scrapy一目了然</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/install.html">安装指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/tutorial.html">Scrapy 教程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/examples.html">实例</a></li>
</ul>
<p class="caption"><span class="caption-text">基本概念</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="commands.html">命令行工具</a></li>
<li class="toctree-l1"><a class="reference internal" href="spiders.html">蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="selectors.html">选择器</a></li>
<li class="toctree-l1"><a class="reference internal" href="items.html">项目</a></li>
<li class="toctree-l1"><a class="reference internal" href="loaders.html">项目加载器</a></li>
<li class="toctree-l1"><a class="reference internal" href="shell.html">Scrapy shell</a></li>
<li class="toctree-l1"><a class="reference internal" href="item-pipeline.html">项目管道</a></li>
<li class="toctree-l1"><a class="reference internal" href="feed-exports.html">Feed 导出</a></li>
<li class="toctree-l1"><a class="reference internal" href="request-response.html">请求和响应</a></li>
<li class="toctree-l1"><a class="reference internal" href="link-extractors.html">链接提取器</a></li>
<li class="toctree-l1"><a class="reference internal" href="settings.html">设置</a></li>
<li class="toctree-l1"><a class="reference internal" href="exceptions.html">例外情况</a></li>
</ul>
<p class="caption"><span class="caption-text">内置服务</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="logging.html">登录</a></li>
<li class="toctree-l1"><a class="reference internal" href="stats.html">统计数据集合</a></li>
<li class="toctree-l1"><a class="reference internal" href="email.html">发送电子邮件</a></li>
<li class="toctree-l1"><a class="reference internal" href="telnetconsole.html">远程登录控制台</a></li>
<li class="toctree-l1"><a class="reference internal" href="webservice.html">Web服务</a></li>
</ul>
<p class="caption"><span class="caption-text">解决具体问题</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../faq.html">常见问题</a></li>
<li class="toctree-l1"><a class="reference internal" href="debug.html">调试spiders</a></li>
<li class="toctree-l1"><a class="reference internal" href="contracts.html">蜘蛛合约</a></li>
<li class="toctree-l1"><a class="reference internal" href="practices.html">常用做法</a></li>
<li class="toctree-l1"><a class="reference internal" href="broad-crawls.html">宽爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="developer-tools.html">使用浏览器的开发人员工具进行抓取</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamic-content.html">选择动态加载的内容</a></li>
<li class="toctree-l1"><a class="reference internal" href="leaks.html">调试内存泄漏</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">下载和处理文件和图像</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#using-the-files-pipeline">使用文件管道</a></li>
<li class="toctree-l2"><a class="reference internal" href="#using-the-images-pipeline">使用图像管道</a></li>
<li class="toctree-l2"><a class="reference internal" href="#enabling-your-media-pipeline">启用媒体管道</a></li>
<li class="toctree-l2"><a class="reference internal" href="#supported-storage">支持的存储</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#file-system-storage">文件系统存储</a></li>
<li class="toctree-l3"><a class="reference internal" href="#ftp-server-storage">FTP服务器存储</a></li>
<li class="toctree-l3"><a class="reference internal" href="#amazon-s3-storage">Amazon S3存储</a></li>
<li class="toctree-l3"><a class="reference internal" href="#google-cloud-storage">谷歌云存储</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#usage-example">使用实例</a></li>
<li class="toctree-l2"><a class="reference internal" href="#additional-features">其他功能</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#file-expiration">文件过期</a></li>
<li class="toctree-l3"><a class="reference internal" href="#thumbnail-generation-for-images">图像的缩略图生成</a></li>
<li class="toctree-l3"><a class="reference internal" href="#filtering-out-small-images">过滤掉小图像</a></li>
<li class="toctree-l3"><a class="reference internal" href="#allowing-redirections">允许重定向</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#module-scrapy.pipelines.files">扩展媒体管道</a></li>
<li class="toctree-l2"><a class="reference internal" href="#custom-images-pipeline-example">自定义图像管道示例</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="deploy.html">部署蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="autothrottle.html">AutoThrottle 扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="benchmarking.html">标杆管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="jobs.html">作业：暂停和恢复爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="coroutines.html">协同程序</a></li>
<li class="toctree-l1"><a class="reference internal" href="asyncio.html">asyncio</a></li>
</ul>
<p class="caption"><span class="caption-text">扩展Scrapy</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="architecture.html">体系结构概述</a></li>
<li class="toctree-l1"><a class="reference internal" href="downloader-middleware.html">下载器中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="spider-middleware.html">蜘蛛中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="extensions.html">扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="api.html">核心API</a></li>
<li class="toctree-l1"><a class="reference internal" href="signals.html">信号</a></li>
<li class="toctree-l1"><a class="reference internal" href="exporters.html">条目导出器</a></li>
</ul>
<p class="caption"><span class="caption-text">其余所有</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../news.html">发行说明</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">为 Scrapy 贡献</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versioning.html">版本控制和API稳定性</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Scrapy</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
        
      <li>下载和处理文件和图像</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="downloading-and-processing-files-and-images">
<span id="topics-media-pipeline"></span><h1>下载和处理文件和图像<a class="headerlink" href="#downloading-and-processing-files-and-images" title="永久链接至标题">¶</a></h1>
<p>Scrapy  可重复使用 <a class="reference internal" href="item-pipeline.html"><span class="doc">item pipelines</span></a> 用于下载附加到特定项目的文件（例如，当您 爬取  产品并希望在本地下载其图像时）。这些管道共享一些功能和结构（我们将它们称为媒体管道），但通常您可以使用文件管道或图像管道。</p>
<p>两条管道都实现了以下功能：</p><script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<ins class="adsbygoogle"
     style="display:block; text-align:center;"
     data-ad-layout="in-article"
     data-ad-format="fluid"
     data-ad-client="ca-pub-1466963416408457"
     data-ad-slot="8850786025"></ins>
<script>
     (adsbygoogle = window.adsbygoogle || []).push({});
</script>
<ul class="simple">
<li><p>避免重新下载最近下载的媒体</p></li>
<li><p>指定存储媒体的位置（文件系统目录、FTP服务器、Amazon S3 bucket、Google云存储bucket）</p></li>
</ul>
<p>图像管道有一些用于处理图像的额外功能：</p>
<ul class="simple">
<li><p>将所有下载的图像转换为通用格式（JPG）和模式（RGB）</p></li>
<li><p>缩略图生成</p></li>
<li><p>检查图像的宽度/高度以确保它们满足最小限制</p></li>
</ul>
<p>这些管道还保留当前正在计划下载的媒体URL的内部队列，并将到达的包含相同媒体的响应连接到该队列。这样可以避免在多个项目共享同一媒体时多次下载同一媒体。</p>
<div class="section" id="using-the-files-pipeline">
<h2>使用文件管道<a class="headerlink" href="#using-the-files-pipeline" title="永久链接至标题">¶</a></h2>
<p>使用时的典型工作流 <code class="xref py py-class docutils literal notranslate"><span class="pre">FilesPipeline</span></code> 像这样：</p>
<ol class="arabic simple">
<li><p>在spider中，您 爬取  一个项目并将所需的URL放入 <code class="docutils literal notranslate"><span class="pre">file_urls</span></code> 字段。</p></li>
<li><p>该项从spider返回并转到项管道。</p></li>
<li><p>当项目到达 <code class="xref py py-class docutils literal notranslate"><span class="pre">FilesPipeline</span></code> ，中的URL <code class="docutils literal notranslate"><span class="pre">file_urls</span></code> 使用标准的Scrapy计划程序和下载程序（这意味着计划程序和下载程序中间软件被重用）来计划下载字段，但具有更高的优先级，在其他页面被抓取之前对其进行处理。该项在特定管道阶段保持“锁定”，直到文件完成下载（或由于某种原因失败）。</p></li>
<li><p>下载文件时，另一个字段 (<code class="docutils literal notranslate"><span class="pre">files</span></code> ）将用结果填充。此字段将包含一个包含有关下载文件的信息的dict列表，例如下载路径、原始的scraped url（取自 <code class="docutils literal notranslate"><span class="pre">file_urls</span></code> 字段），文件校验和和和文件状态。列表中的文件 <code class="docutils literal notranslate"><span class="pre">files</span></code> 字段将保留与原始字段相同的顺序 <code class="docutils literal notranslate"><span class="pre">file_urls</span></code> 字段。如果某些文件下载失败，将记录一个错误，并且该文件不会出现在 <code class="docutils literal notranslate"><span class="pre">files</span></code> 字段。</p></li>
</ol>
</div>
<div class="section" id="using-the-images-pipeline">
<h2>使用图像管道<a class="headerlink" href="#using-the-images-pipeline" title="永久链接至标题">¶</a></h2>
<p>使用 <a class="reference internal" href="#scrapy.pipelines.images.ImagesPipeline" title="scrapy.pipelines.images.ImagesPipeline"><code class="xref py py-class docutils literal notranslate"><span class="pre">ImagesPipeline</span></code></a> 很像使用 <code class="xref py py-class docutils literal notranslate"><span class="pre">FilesPipeline</span></code> ，但使用的默认字段名不同：您使用 <code class="docutils literal notranslate"><span class="pre">image_urls</span></code> 对于项目的图像URL，它将填充 <code class="docutils literal notranslate"><span class="pre">images</span></code> 有关下载图像的信息字段。</p>
<p>使用 <a class="reference internal" href="#scrapy.pipelines.images.ImagesPipeline" title="scrapy.pipelines.images.ImagesPipeline"><code class="xref py py-class docutils literal notranslate"><span class="pre">ImagesPipeline</span></code></a> 对于图像文件，您可以配置一些额外的功能，如生成缩略图和根据图像大小过滤图像。</p>
<p>图像管道使用 <a class="reference external" href="https://github.com/python-pillow/Pillow">Pillow</a> 对于缩略图和将图像规范化为jpeg/rgb格式，因此需要安装此库才能使用它。 <a class="reference external" href="http://www.pythonware.com/products/pil/">Python Imaging Library</a> （PIL）在大多数情况下也应该有效，但它在某些设置中会引起问题，因此我们建议使用 <a class="reference external" href="https://github.com/python-pillow/Pillow">Pillow</a> 而不是皮尔。</p>
</div>
<div class="section" id="enabling-your-media-pipeline">
<span id="topics-media-pipeline-enabling"></span><h2>启用媒体管道<a class="headerlink" href="#enabling-your-media-pipeline" title="永久链接至标题">¶</a></h2>
<span class="target" id="std-setting-IMAGES_STORE"><span id="std:setting-IMAGES_STORE"></span></span><p id="std-setting-FILES_STORE"><span id="std:setting-FILES_STORE"></span>要启用媒体管道，必须首先将其添加到项目中 <a class="reference internal" href="settings.html#std-setting-ITEM_PIPELINES"><code class="xref std std-setting docutils literal notranslate"><span class="pre">ITEM_PIPELINES</span></code></a> 设置。</p>
<p>对于图像管道，请使用：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ITEM_PIPELINES</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;scrapy.pipelines.images.ImagesPipeline&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span>
</pre></div>
</div>
<p>对于文件管道，请使用：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ITEM_PIPELINES</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;scrapy.pipelines.files.FilesPipeline&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>您还可以同时使用文件和图像管道。</p>
</div>
<p>然后，将目标存储设置配置为用于存储下载的图像的有效值。否则，管道将保持禁用状态，即使将其包含在 <a class="reference internal" href="settings.html#std-setting-ITEM_PIPELINES"><code class="xref std std-setting docutils literal notranslate"><span class="pre">ITEM_PIPELINES</span></code></a> 设置。</p>
<p>对于文件管道，设置 <a class="reference internal" href="#std-setting-FILES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE</span></code></a> 设置：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">FILES_STORE</span> <span class="o">=</span> <span class="s1">&#39;/path/to/valid/dir&#39;</span>
</pre></div>
</div>
<p>对于图像管道，设置 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 设置：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_STORE</span> <span class="o">=</span> <span class="s1">&#39;/path/to/valid/dir&#39;</span>
</pre></div>
</div>
</div>
<div class="section" id="supported-storage">
<h2>支持的存储<a class="headerlink" href="#supported-storage" title="永久链接至标题">¶</a></h2>
<div class="section" id="file-system-storage">
<h3>文件系统存储<a class="headerlink" href="#file-system-storage" title="永久链接至标题">¶</a></h3>
<p>文件使用 <a class="reference external" href="https://en.wikipedia.org/wiki/SHA_hash_functions">SHA1 hash</a> 文件名的URL。</p>
<p>例如，以下图像URL:：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">www</span><span class="o">.</span><span class="n">example</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">image</span><span class="o">.</span><span class="n">jpg</span>
</pre></div>
</div>
<p>谁的 <code class="docutils literal notranslate"><span class="pre">SHA1</span> <span class="pre">hash</span></code> 是：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">3</span><span class="n">afec3b4765f8f0a07b78f98c07b83f013567a0a</span>
</pre></div>
</div>
<p>将下载并存储在以下文件中：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">IMAGES_STORE</span><span class="o">&gt;/</span><span class="n">full</span><span class="o">/</span><span class="mi">3</span><span class="n">afec3b4765f8f0a07b78f98c07b83f013567a0a</span><span class="o">.</span><span class="n">jpg</span>
</pre></div>
</div>
<p>在哪里？</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">&lt;IMAGES_STORE&gt;</span></code> 目录是否在中定义？ <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 图像管道的设置。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">full</span></code> 是一个子目录，用于从缩略图中分离完整图像（如果使用）。有关详细信息，请参阅 <a class="reference internal" href="#topics-images-thumbnails"><span class="std std-ref">图像的缩略图生成</span></a> .</p></li>
</ul>
</div>
<div class="section" id="ftp-server-storage">
<span id="media-pipeline-ftp"></span><h3>FTP服务器存储<a class="headerlink" href="#ftp-server-storage" title="永久链接至标题">¶</a></h3>
<div class="versionadded">
<p><span class="versionmodified added">2.0 新版功能.</span></p>
</div>
<p><a class="reference internal" href="#std-setting-FILES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 可以指向FTP服务器。Scrapy会自动将文件上传到服务器。</p>
<p><a class="reference internal" href="#std-setting-FILES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 应以下列形式之一书写：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ftp</span><span class="p">:</span><span class="o">//</span><span class="n">username</span><span class="p">:</span><span class="n">password</span><span class="nd">@address</span><span class="p">:</span><span class="n">port</span><span class="o">/</span><span class="n">path</span>
<span class="n">ftp</span><span class="p">:</span><span class="o">//</span><span class="n">address</span><span class="p">:</span><span class="n">port</span><span class="o">/</span><span class="n">path</span>
</pre></div>
</div>
<p>如果 <code class="docutils literal notranslate"><span class="pre">username</span></code> 和 <code class="docutils literal notranslate"><span class="pre">password</span></code> 不提供，它们是从 <a class="reference internal" href="settings.html#std-setting-FTP_USER"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FTP_USER</span></code></a> 和 <a class="reference internal" href="settings.html#std-setting-FTP_PASSWORD"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FTP_PASSWORD</span></code></a> 分别设置。</p>
<p>FTP支持两种不同的连接模式：主动或被动。Scrapy默认使用被动连接模式。要使用活动连接模式，请设置 <a class="reference internal" href="feed-exports.html#std-setting-FEED_STORAGE_FTP_ACTIVE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FEED_STORAGE_FTP_ACTIVE</span></code></a> 设置为 <code class="docutils literal notranslate"><span class="pre">True</span></code> .</p>
</div>
<div class="section" id="amazon-s3-storage">
<h3>Amazon S3存储<a class="headerlink" href="#amazon-s3-storage" title="永久链接至标题">¶</a></h3>
<span class="target" id="std-setting-FILES_STORE_S3_ACL"><span id="std:setting-FILES_STORE_S3_ACL"></span></span><p id="std-setting-IMAGES_STORE_S3_ACL"><span id="std:setting-IMAGES_STORE_S3_ACL"></span><a class="reference internal" href="#std-setting-FILES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 可以表示AmazonS3存储桶。Scrapy会自动将文件上传到bucket。</p>
<p>例如，这是一个有效的 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 价值：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_STORE</span> <span class="o">=</span> <span class="s1">&#39;s3://bucket/images&#39;</span>
</pre></div>
</div>
<p>您可以修改用于存储文件的访问控制列表（ACL）策略，该策略由 <a class="reference internal" href="#std-setting-FILES_STORE_S3_ACL"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE_S3_ACL</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_STORE_S3_ACL"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE_S3_ACL</span></code></a> 设置。默认情况下，acl设置为 <code class="docutils literal notranslate"><span class="pre">private</span></code> . 要使文件公开，请使用 <code class="docutils literal notranslate"><span class="pre">public-read</span></code> 政策：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_STORE_S3_ACL</span> <span class="o">=</span> <span class="s1">&#39;public-read&#39;</span>
</pre></div>
</div>
<p>有关详细信息，请参阅 <a class="reference external" href="https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl">canned ACLs</a> 在AmazonS3开发者指南中。</p>
<p>因为下流的东西 <code class="docutils literal notranslate"><span class="pre">botocore</span></code> 在内部，您还可以使用其他S3，如存储。像自我托管这样的存储 <a class="reference external" href="https://github.com/minio/minio">Minio</a> 或 <a class="reference external" href="https://s3.scality.com/">s3.scality</a> . 您所需要做的就是在Scrapy设置中设置端点选项：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">AWS_ENDPOINT_URL</span> <span class="o">=</span> <span class="s1">&#39;http://minio.example.com:9000&#39;</span>
</pre></div>
</div>
<p>对于自托管，您可能觉得不需要使用SSL，也不需要验证SSL连接：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">AWS_USE_SSL</span> <span class="o">=</span> <span class="kc">False</span> <span class="c1"># or True (None by default)</span>
<span class="n">AWS_VERIFY</span> <span class="o">=</span> <span class="kc">False</span> <span class="c1"># or True (None by default)</span>
</pre></div>
</div>
</div>
<div class="section" id="google-cloud-storage">
<span id="media-pipeline-gcs"></span><h3>谷歌云存储<a class="headerlink" href="#google-cloud-storage" title="永久链接至标题">¶</a></h3>
<span class="target" id="std-setting-FILES_STORE_GCS_ACL"><span id="std:setting-FILES_STORE_GCS_ACL"></span></span><p id="std-setting-IMAGES_STORE_GCS_ACL"><span id="std:setting-IMAGES_STORE_GCS_ACL"></span><a class="reference internal" href="#std-setting-FILES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 可以表示一个谷歌云存储桶。Scrapy会自动将文件上传到bucket。（需要 <a class="reference external" href="https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python">google-cloud-storage</a> ）</p>
<p>例如，这些是有效的 <a class="reference internal" href="#std-setting-IMAGES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE</span></code></a> 和 <a class="reference internal" href="settings.html#std-setting-GCS_PROJECT_ID"><code class="xref std std-setting docutils literal notranslate"><span class="pre">GCS_PROJECT_ID</span></code></a> 设置：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_STORE</span> <span class="o">=</span> <span class="s1">&#39;gs://bucket/images/&#39;</span>
<span class="n">GCS_PROJECT_ID</span> <span class="o">=</span> <span class="s1">&#39;project_id&#39;</span>
</pre></div>
</div>
<p>有关身份验证的信息，请参见 <a class="reference external" href="https://cloud.google.com/docs/authentication/production">documentation</a> .</p>
<p>您可以修改用于存储文件的访问控制列表（ACL）策略，该策略由 <a class="reference internal" href="#std-setting-FILES_STORE_GCS_ACL"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE_GCS_ACL</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_STORE_GCS_ACL"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_STORE_GCS_ACL</span></code></a> 设置。默认情况下，acl设置为 <code class="docutils literal notranslate"><span class="pre">''</span></code> （空字符串）这意味着云存储将bucket的默认对象acl应用于该对象。要使文件公开，请使用 <code class="docutils literal notranslate"><span class="pre">publicRead</span></code> 政策：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_STORE_GCS_ACL</span> <span class="o">=</span> <span class="s1">&#39;publicRead&#39;</span>
</pre></div>
</div>
<p>有关详细信息，请参阅 <a class="reference external" href="https://cloud.google.com/storage/docs/access-control/lists#predefined-acl">Predefined ACLs</a> 在谷歌云平台开发者指南中。</p>
</div>
</div>
<div class="section" id="usage-example">
<h2>使用实例<a class="headerlink" href="#usage-example" title="永久链接至标题">¶</a></h2>
<span class="target" id="std-setting-FILES_URLS_FIELD"><span id="std:setting-FILES_URLS_FIELD"></span></span><span class="target" id="std-setting-FILES_RESULT_FIELD"><span id="std:setting-FILES_RESULT_FIELD"></span></span><span class="target" id="std-setting-IMAGES_URLS_FIELD"><span id="std:setting-IMAGES_URLS_FIELD"></span></span><p id="std-setting-IMAGES_RESULT_FIELD"><span id="std:setting-IMAGES_RESULT_FIELD"></span>为了使用媒体管道，首先 <a class="reference internal" href="#topics-media-pipeline-enabling"><span class="std std-ref">enable it</span></a> .</p>
<p>那么，如果蜘蛛返回一个 <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">item object</span></a> 使用URL字段 (<code class="docutils literal notranslate"><span class="pre">file_urls</span></code> 或 <code class="docutils literal notranslate"><span class="pre">image_urls</span></code> ，对于文件或图像管道），管道会将结果放在相应的字段下 (<code class="docutils literal notranslate"><span class="pre">files</span></code> 或 <code class="docutils literal notranslate"><span class="pre">images</span></code> ）</p>
<p>使用时 <a class="reference internal" href="items.html#item-types"><span class="std std-ref">item types</span></a> 对于预先定义的字段，必须同时定义url字段和results字段。例如，当使用图像管道时，项必须同时定义 <code class="docutils literal notranslate"><span class="pre">image_urls</span></code> 以及 <code class="docutils literal notranslate"><span class="pre">images</span></code> 字段。例如，使用 <a class="reference internal" href="items.html#scrapy.item.Item" title="scrapy.item.Item"><code class="xref py py-class docutils literal notranslate"><span class="pre">Item</span></code></a> 班级：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>

<span class="k">class</span> <span class="nc">MyItem</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Item</span><span class="p">):</span>
    <span class="c1"># ... other item fields ...</span>
    <span class="n">image_urls</span> <span class="o">=</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Field</span><span class="p">()</span>
    <span class="n">images</span> <span class="o">=</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Field</span><span class="p">()</span>
</pre></div>
</div>
<p>如果要对URL键或结果键使用其他字段名，也可以重写它。</p>
<p>对于文件管道，设置 <a class="reference internal" href="#std-setting-FILES_URLS_FIELD"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_URLS_FIELD</span></code></a> 和/或 <a class="reference internal" href="#std-setting-FILES_RESULT_FIELD"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_RESULT_FIELD</span></code></a> 设置：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">FILES_URLS_FIELD</span> <span class="o">=</span> <span class="s1">&#39;field_name_for_your_files_urls&#39;</span>
<span class="n">FILES_RESULT_FIELD</span> <span class="o">=</span> <span class="s1">&#39;field_name_for_your_processed_files&#39;</span>
</pre></div>
</div>
<p>对于图像管道，设置 <a class="reference internal" href="#std-setting-IMAGES_URLS_FIELD"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_URLS_FIELD</span></code></a> 和/或 <a class="reference internal" href="#std-setting-IMAGES_RESULT_FIELD"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_RESULT_FIELD</span></code></a> 设置：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_URLS_FIELD</span> <span class="o">=</span> <span class="s1">&#39;field_name_for_your_images_urls&#39;</span>
<span class="n">IMAGES_RESULT_FIELD</span> <span class="o">=</span> <span class="s1">&#39;field_name_for_your_processed_images&#39;</span>
</pre></div>
</div>
<p>如果您需要更复杂的内容，并且想要覆盖自定义管道行为，请参见 <a class="reference internal" href="#topics-media-pipeline-override"><span class="std std-ref">扩展媒体管道</span></a> .</p>
<p>如果有多个图像管道继承自ImagePipeline，并且希望在不同的管道中具有不同的设置，则可以设置以管道类的大写名称开头的设置键。例如，如果您的管道名为mypipeline，并且您希望有自定义图像URL字段，那么您可以定义设置mypipeline图像URL字段，并且将使用自定义设置。</p>
</div>
<div class="section" id="additional-features">
<h2>其他功能<a class="headerlink" href="#additional-features" title="永久链接至标题">¶</a></h2>
<div class="section" id="file-expiration">
<h3>文件过期<a class="headerlink" href="#file-expiration" title="永久链接至标题">¶</a></h3>
<span class="target" id="std-setting-IMAGES_EXPIRES"><span id="std:setting-IMAGES_EXPIRES"></span></span><p id="std-setting-FILES_EXPIRES"><span id="std:setting-FILES_EXPIRES"></span>图像管道避免下载最近下载的文件。要调整此保留延迟，请使用 <a class="reference internal" href="#std-setting-FILES_EXPIRES"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_EXPIRES</span></code></a> 设置（或） <a class="reference internal" href="#std-setting-IMAGES_EXPIRES"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_EXPIRES</span></code></a> ，对于图像管道），指定延迟天数：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># 120 days of delay for files expiration</span>
<span class="n">FILES_EXPIRES</span> <span class="o">=</span> <span class="mi">120</span>

<span class="c1"># 30 days of delay for images expiration</span>
<span class="n">IMAGES_EXPIRES</span> <span class="o">=</span> <span class="mi">30</span>
</pre></div>
</div>
<p>两种设置的默认值都是90天。</p>
<p>如果您有子类filespine的管道，并且希望对其进行不同的设置，则可以设置以大写类名开头的设置键。例如，给定名为MyPipeline的管道类，您可以设置设置键：</p>
<blockquote>
<div><p>mypipeline_files_expires=180</p>
</div></blockquote>
<p>管道类MyPipeline的过期时间设置为180。</p>
</div>
<div class="section" id="thumbnail-generation-for-images">
<span id="topics-images-thumbnails"></span><h3>图像的缩略图生成<a class="headerlink" href="#thumbnail-generation-for-images" title="永久链接至标题">¶</a></h3>
<p>图像管道可以自动创建下载图像的缩略图。</p>
<p id="std-setting-IMAGES_THUMBS"><span id="std:setting-IMAGES_THUMBS"></span>要使用此功能，必须设置 <a class="reference internal" href="#std-setting-IMAGES_THUMBS"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_THUMBS</span></code></a> 到一个字典，其中键是缩略图名称，值是它们的尺寸。</p>
<p>例如：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_THUMBS</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&#39;small&#39;</span><span class="p">:</span> <span class="p">(</span><span class="mi">50</span><span class="p">,</span> <span class="mi">50</span><span class="p">),</span>
    <span class="s1">&#39;big&#39;</span><span class="p">:</span> <span class="p">(</span><span class="mi">270</span><span class="p">,</span> <span class="mi">270</span><span class="p">),</span>
<span class="p">}</span>
</pre></div>
</div>
<p>使用此功能时，图像管道将使用以下格式创建每个指定大小的缩略图：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">IMAGES_STORE</span><span class="o">&gt;/</span><span class="n">thumbs</span><span class="o">/&lt;</span><span class="n">size_name</span><span class="o">&gt;/&lt;</span><span class="n">image_id</span><span class="o">&gt;.</span><span class="n">jpg</span>
</pre></div>
</div>
<p>在哪里？</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">&lt;size_name&gt;</span></code> 是在 <a class="reference internal" href="#std-setting-IMAGES_THUMBS"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_THUMBS</span></code></a> 字典键 (<code class="docutils literal notranslate"><span class="pre">small</span></code> ， <code class="docutils literal notranslate"><span class="pre">big</span></code> 等）</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">&lt;image_id&gt;</span></code> 是 <a class="reference external" href="https://en.wikipedia.org/wiki/SHA_hash_functions">SHA1 hash</a> 图像URL的</p></li>
</ul>
<p>存储图像文件的示例 <code class="docutils literal notranslate"><span class="pre">small</span></code> 和 <code class="docutils literal notranslate"><span class="pre">big</span></code> 缩略图名称：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">IMAGES_STORE</span><span class="o">&gt;/</span><span class="n">full</span><span class="o">/</span><span class="mi">63</span><span class="n">bbfea82b8880ed33cdb762aa11fab722a90a24</span><span class="o">.</span><span class="n">jpg</span>
<span class="o">&lt;</span><span class="n">IMAGES_STORE</span><span class="o">&gt;/</span><span class="n">thumbs</span><span class="o">/</span><span class="n">small</span><span class="o">/</span><span class="mi">63</span><span class="n">bbfea82b8880ed33cdb762aa11fab722a90a24</span><span class="o">.</span><span class="n">jpg</span>
<span class="o">&lt;</span><span class="n">IMAGES_STORE</span><span class="o">&gt;/</span><span class="n">thumbs</span><span class="o">/</span><span class="n">big</span><span class="o">/</span><span class="mi">63</span><span class="n">bbfea82b8880ed33cdb762aa11fab722a90a24</span><span class="o">.</span><span class="n">jpg</span>
</pre></div>
</div>
<p>第一个是从网站下载的完整图像。</p>
</div>
<div class="section" id="filtering-out-small-images">
<h3>过滤掉小图像<a class="headerlink" href="#filtering-out-small-images" title="永久链接至标题">¶</a></h3>
<span class="target" id="std-setting-IMAGES_MIN_HEIGHT"><span id="std:setting-IMAGES_MIN_HEIGHT"></span></span><p id="std-setting-IMAGES_MIN_WIDTH"><span id="std:setting-IMAGES_MIN_WIDTH"></span>使用图像管道时，可以删除过小的图像，方法是在 <a class="reference internal" href="#std-setting-IMAGES_MIN_HEIGHT"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_MIN_HEIGHT</span></code></a> 和 <a class="reference internal" href="#std-setting-IMAGES_MIN_WIDTH"><code class="xref std std-setting docutils literal notranslate"><span class="pre">IMAGES_MIN_WIDTH</span></code></a> 设置。</p>
<p>例如：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">IMAGES_MIN_HEIGHT</span> <span class="o">=</span> <span class="mi">110</span>
<span class="n">IMAGES_MIN_WIDTH</span> <span class="o">=</span> <span class="mi">110</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>大小约束根本不影响缩略图的生成。</p>
</div>
<p>可以只设置一个大小约束或同时设置两个大小约束。当同时设置这两种尺寸时，将只保存满足这两种最小尺寸的图像。对于上面的示例，大小为（105 x 105）或（105 x 200）或（200 x 105）的图像都将被删除，因为至少有一个维度比约束短。</p>
<p>默认情况下，没有大小约束，因此所有图像都会被处理。</p>
</div>
<div class="section" id="allowing-redirections">
<h3>允许重定向<a class="headerlink" href="#allowing-redirections" title="永久链接至标题">¶</a></h3>
<p id="std-setting-MEDIA_ALLOW_REDIRECTS"><span id="std:setting-MEDIA_ALLOW_REDIRECTS"></span>默认情况下，媒体管道忽略重定向，即HTTP重定向到媒体文件URL请求将意味着媒体下载失败。</p>
<p>要处理媒体重定向，请将此设置设置为 <code class="docutils literal notranslate"><span class="pre">True</span></code> ：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">MEDIA_ALLOW_REDIRECTS</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="module-scrapy.pipelines.files">
<span id="extending-the-media-pipelines"></span><span id="topics-media-pipeline-override"></span><h2>扩展媒体管道<a class="headerlink" href="#module-scrapy.pipelines.files" title="永久链接至标题">¶</a></h2>
<p>请参见以下自定义文件管道中可以重写的方法：</p>
<dl class="py class">
<dt id="scrapy.pipelines.files.FilesPipeline">
<em class="property">class </em><code class="sig-prename descclassname">scrapy.pipelines.files.</code><code class="sig-name descname">FilesPipeline</code><a class="headerlink" href="#scrapy.pipelines.files.FilesPipeline" title="永久链接至目标">¶</a></dt>
<dd><dl class="py method">
<dt id="scrapy.pipelines.files.FilesPipeline.file_path">
<code class="sig-name descname">file_path</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">self</span></em>, <em class="sig-param"><span class="n">request</span></em>, <em class="sig-param"><span class="n">response</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="n">info</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="o">*</span></em>, <em class="sig-param"><span class="n">item</span><span class="o">=</span><span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#scrapy.pipelines.files.FilesPipeline.file_path" title="永久链接至目标">¶</a></dt>
<dd><p>每个下载的项调用一次此方法。它返回源于指定的 <a class="reference internal" href="request-response.html#scrapy.http.Response" title="scrapy.http.Response"><code class="xref py py-class docutils literal notranslate"><span class="pre">response</span></code></a> .</p>
<p>除了 <code class="docutils literal notranslate"><span class="pre">response</span></code> ，此方法接收原始 <code class="xref py py-class docutils literal notranslate"><span class="pre">request</span></code> ， <code class="xref py py-class docutils literal notranslate"><span class="pre">info</span></code> 和 <a class="reference internal" href="items.html#scrapy.item.Item" title="scrapy.item.Item"><code class="xref py py-class docutils literal notranslate"><span class="pre">item</span></code></a></p>
<p>可以重写此方法以自定义每个文件的下载路径。</p>
<p>例如，如果文件URL以常规路径结束（例如 <code class="docutils literal notranslate"><span class="pre">https://example.com/a/b/c/foo.png</span></code> ，您可以使用以下方法将所有文件下载到 <code class="docutils literal notranslate"><span class="pre">files</span></code> 文件夹及其原始文件名（例如 <code class="docutils literal notranslate"><span class="pre">files/foo.png</span></code> ）：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>

<span class="kn">from</span> <span class="nn">scrapy.pipelines.files</span> <span class="kn">import</span> <span class="n">FilesPipeline</span>

<span class="k">class</span> <span class="nc">MyFilesPipeline</span><span class="p">(</span><span class="n">FilesPipeline</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">file_path</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">response</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">info</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">item</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="k">return</span> <span class="s1">&#39;files/&#39;</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">urlparse</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
</pre></div>
</div>
<p>同样，您可以使用 <code class="docutils literal notranslate"><span class="pre">item</span></code> 根据某些项属性确定文件路径。</p>
<p>默认情况下 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.file_path" title="scrapy.pipelines.files.FilesPipeline.file_path"><code class="xref py py-meth docutils literal notranslate"><span class="pre">file_path()</span></code></a> 方法返回 <code class="docutils literal notranslate"><span class="pre">full/&lt;request</span> <span class="pre">URL</span> <span class="pre">hash&gt;.&lt;extension&gt;</span></code> .</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.pipelines.files.FilesPipeline.get_media_requests">
<code class="sig-name descname">get_media_requests</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">item</span></em>, <em class="sig-param"><span class="n">info</span></em><span class="sig-paren">)</span><a class="headerlink" href="#scrapy.pipelines.files.FilesPipeline.get_media_requests" title="永久链接至目标">¶</a></dt>
<dd><p>如工作流上所示，管道将从项目中获取要下载的图像的URL。为此，可以重写 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.get_media_requests" title="scrapy.pipelines.files.FilesPipeline.get_media_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get_media_requests()</span></code></a> 方法并返回每个文件的请求URL:：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">itemadapter</span> <span class="kn">import</span> <span class="n">ItemAdapter</span>

<span class="k">def</span> <span class="nf">get_media_requests</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span>
    <span class="n">adapter</span> <span class="o">=</span> <span class="n">ItemAdapter</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">file_url</span> <span class="ow">in</span> <span class="n">adapter</span><span class="p">[</span><span class="s1">&#39;file_urls&#39;</span><span class="p">]:</span>
        <span class="k">yield</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">file_url</span><span class="p">)</span>
</pre></div>
</div>
<p>这些请求将由管道处理，下载完成后，结果将发送到 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="scrapy.pipelines.files.FilesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">item_completed()</span></code></a> 方法，作为2元素元组的列表。每个元组将包含 <code class="docutils literal notranslate"><span class="pre">(success,</span> <span class="pre">file_info_or_error)</span></code> 在哪里？</p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">success</span></code> 是一个布尔值，它是 <code class="docutils literal notranslate"><span class="pre">True</span></code> 如果图像下载成功或 <code class="docutils literal notranslate"><span class="pre">False</span></code> 如果因为某种原因失败了</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">file_info_or_error</span></code> 是包含以下键的dict（如果成功是 <code class="docutils literal notranslate"><span class="pre">True</span></code> 或A <a class="reference external" href="https://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html" title="(在 Twisted v2.0)"><code class="xref py py-exc docutils literal notranslate"><span class="pre">Failure</span></code></a> 如果有问题的话。</p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">url</span></code> -从中下载文件的URL。这是从 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.get_media_requests" title="scrapy.pipelines.files.FilesPipeline.get_media_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get_media_requests()</span></code></a> 方法。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">path</span></code> -路径（相对于 <a class="reference internal" href="#std-setting-FILES_STORE"><code class="xref std std-setting docutils literal notranslate"><span class="pre">FILES_STORE</span></code></a> ）文件的存储位置</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">checksum</span></code> -A <a class="reference external" href="https://en.wikipedia.org/wiki/MD5">MD5 hash</a> 图像内容的</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">status</span></code> -文件状态指示。</p>
<div class="versionadded">
<p><span class="versionmodified added">2.2 新版功能.</span></p>
</div>
<p>它可以是以下类型之一：</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">downloaded</span></code> -文件已下载。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">uptodate</span></code> -根据文件过期策略，文件未下载，因为它是最近下载的。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">cached</span></code> -文件已被共享同一文件的另一个项目计划下载。</p></li>
</ul>
</li>
</ul>
</li>
</ul>
<p>接收的元组列表 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="scrapy.pipelines.files.FilesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">item_completed()</span></code></a> 保证保留从 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.get_media_requests" title="scrapy.pipelines.files.FilesPipeline.get_media_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get_media_requests()</span></code></a> 方法。</p>
<p>以下是 <code class="docutils literal notranslate"><span class="pre">results</span></code> 论点：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[(</span><span class="kc">True</span><span class="p">,</span>
  <span class="p">{</span><span class="s1">&#39;checksum&#39;</span><span class="p">:</span> <span class="s1">&#39;2b00042f7481c7b056c4b410d28f33cf&#39;</span><span class="p">,</span>
   <span class="s1">&#39;path&#39;</span><span class="p">:</span> <span class="s1">&#39;full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg&#39;</span><span class="p">,</span>
   <span class="s1">&#39;url&#39;</span><span class="p">:</span> <span class="s1">&#39;http://www.example.com/files/product1.pdf&#39;</span><span class="p">,</span>
   <span class="s1">&#39;status&#39;</span><span class="p">:</span> <span class="s1">&#39;downloaded&#39;</span><span class="p">}),</span>
 <span class="p">(</span><span class="kc">False</span><span class="p">,</span>
  <span class="n">Failure</span><span class="p">(</span><span class="o">...</span><span class="p">))]</span>
</pre></div>
</div>
<p>默认情况下 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.get_media_requests" title="scrapy.pipelines.files.FilesPipeline.get_media_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get_media_requests()</span></code></a> 方法返回 <code class="docutils literal notranslate"><span class="pre">None</span></code> 这意味着该项目没有可下载的文件。</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.pipelines.files.FilesPipeline.item_completed">
<code class="sig-name descname">item_completed</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">results</span></em>, <em class="sig-param"><span class="n">item</span></em>, <em class="sig-param"><span class="n">info</span></em><span class="sig-paren">)</span><a class="headerlink" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="永久链接至目标">¶</a></dt>
<dd><p>这个 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="scrapy.pipelines.files.FilesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">FilesPipeline.item_completed()</span></code></a> 当单个项的所有文件请求都已完成时调用的方法（要么已完成下载，要么由于某种原因失败）。</p>
<p>这个 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="scrapy.pipelines.files.FilesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">item_completed()</span></code></a> 方法必须返回将发送到后续项管道阶段的输出，因此必须返回（或删除）该项，就像在任何管道中一样。</p>
<p>下面是一个 <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="scrapy.pipelines.files.FilesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">item_completed()</span></code></a> 方法，将下载的文件路径（传入结果）存储在 <code class="docutils literal notranslate"><span class="pre">file_paths</span></code> 项目字段，如果该项目不包含任何文件，则将其删除：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">itemadapter</span> <span class="kn">import</span> <span class="n">ItemAdapter</span>
<span class="kn">from</span> <span class="nn">scrapy.exceptions</span> <span class="kn">import</span> <span class="n">DropItem</span>

<span class="k">def</span> <span class="nf">item_completed</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">results</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span>
    <span class="n">file_paths</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="s1">&#39;path&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">ok</span><span class="p">,</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">ok</span><span class="p">]</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="n">file_paths</span><span class="p">:</span>
        <span class="k">raise</span> <span class="n">DropItem</span><span class="p">(</span><span class="s2">&quot;Item contains no files&quot;</span><span class="p">)</span>
    <span class="n">adapter</span> <span class="o">=</span> <span class="n">ItemAdapter</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
    <span class="n">adapter</span><span class="p">[</span><span class="s1">&#39;file_paths&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">file_paths</span>
    <span class="k">return</span> <span class="n">item</span>
</pre></div>
</div>
<p>默认情况下， <a class="reference internal" href="#scrapy.pipelines.files.FilesPipeline.item_completed" title="scrapy.pipelines.files.FilesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">item_completed()</span></code></a> 方法返回项。</p>
</dd></dl>

</dd></dl>

<span class="target" id="module-scrapy.pipelines.images"></span><p>请参见以下自定义图像管道中可以覆盖的方法：</p>
<dl class="py class">
<dt id="scrapy.pipelines.images.ImagesPipeline">
<em class="property">class </em><code class="sig-prename descclassname">scrapy.pipelines.images.</code><code class="sig-name descname">ImagesPipeline</code><a class="headerlink" href="#scrapy.pipelines.images.ImagesPipeline" title="永久链接至目标">¶</a></dt>
<dd><blockquote>
<div><p>这个 <a class="reference internal" href="#scrapy.pipelines.images.ImagesPipeline" title="scrapy.pipelines.images.ImagesPipeline"><code class="xref py py-class docutils literal notranslate"><span class="pre">ImagesPipeline</span></code></a> 是 <code class="xref py py-class docutils literal notranslate"><span class="pre">FilesPipeline</span></code> ，自定义字段名并为图像添加自定义行为。</p>
</div></blockquote>
<dl class="py method">
<dt id="scrapy.pipelines.images.ImagesPipeline.file_path">
<code class="sig-name descname">file_path</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">self</span></em>, <em class="sig-param"><span class="n">request</span></em>, <em class="sig-param"><span class="n">response</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="n">info</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="o">*</span></em>, <em class="sig-param"><span class="n">item</span><span class="o">=</span><span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#scrapy.pipelines.images.ImagesPipeline.file_path" title="永久链接至目标">¶</a></dt>
<dd><p>每个下载的项调用一次此方法。它返回源于指定的 <a class="reference internal" href="request-response.html#scrapy.http.Response" title="scrapy.http.Response"><code class="xref py py-class docutils literal notranslate"><span class="pre">response</span></code></a> .</p>
<p>除了 <code class="docutils literal notranslate"><span class="pre">response</span></code> ，此方法接收原始 <code class="xref py py-class docutils literal notranslate"><span class="pre">request</span></code> ， <code class="xref py py-class docutils literal notranslate"><span class="pre">info</span></code> 和 <a class="reference internal" href="items.html#scrapy.item.Item" title="scrapy.item.Item"><code class="xref py py-class docutils literal notranslate"><span class="pre">item</span></code></a></p>
<p>可以重写此方法以自定义每个文件的下载路径。</p>
<p>例如，如果文件URL以常规路径结束（例如 <code class="docutils literal notranslate"><span class="pre">https://example.com/a/b/c/foo.png</span></code> ，您可以使用以下方法将所有文件下载到 <code class="docutils literal notranslate"><span class="pre">files</span></code> 文件夹及其原始文件名（例如 <code class="docutils literal notranslate"><span class="pre">files/foo.png</span></code> ）：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>

<span class="kn">from</span> <span class="nn">scrapy.pipelines.images</span> <span class="kn">import</span> <span class="n">ImagesPipeline</span>

<span class="k">class</span> <span class="nc">MyImagesPipeline</span><span class="p">(</span><span class="n">ImagesPipeline</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">file_path</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">response</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">info</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">item</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="k">return</span> <span class="s1">&#39;files/&#39;</span> <span class="o">+</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">urlparse</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
</pre></div>
</div>
<p>同样，您可以使用 <code class="docutils literal notranslate"><span class="pre">item</span></code> 根据某些项属性确定文件路径。</p>
<p>默认情况下 <a class="reference internal" href="#scrapy.pipelines.images.ImagesPipeline.file_path" title="scrapy.pipelines.images.ImagesPipeline.file_path"><code class="xref py py-meth docutils literal notranslate"><span class="pre">file_path()</span></code></a> 方法返回 <code class="docutils literal notranslate"><span class="pre">full/&lt;request</span> <span class="pre">URL</span> <span class="pre">hash&gt;.&lt;extension&gt;</span></code> .</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.pipelines.images.ImagesPipeline.get_media_requests">
<code class="sig-name descname">get_media_requests</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">item</span></em>, <em class="sig-param"><span class="n">info</span></em><span class="sig-paren">)</span><a class="headerlink" href="#scrapy.pipelines.images.ImagesPipeline.get_media_requests" title="永久链接至目标">¶</a></dt>
<dd><p>工作方式与 <code class="xref py py-meth docutils literal notranslate"><span class="pre">FilesPipeline.get_media_requests()</span></code> 方法，但对图像URL使用不同的字段名。</p>
<p>必须返回每个图像URL的请求。</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.pipelines.images.ImagesPipeline.item_completed">
<code class="sig-name descname">item_completed</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">results</span></em>, <em class="sig-param"><span class="n">item</span></em>, <em class="sig-param"><span class="n">info</span></em><span class="sig-paren">)</span><a class="headerlink" href="#scrapy.pipelines.images.ImagesPipeline.item_completed" title="永久链接至目标">¶</a></dt>
<dd><p>这个 <a class="reference internal" href="#scrapy.pipelines.images.ImagesPipeline.item_completed" title="scrapy.pipelines.images.ImagesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">ImagesPipeline.item_completed()</span></code></a> 当一个项目的所有图像请求都已完成时（要么已完成下载，要么由于某种原因失败），将调用方法。</p>
<p>工作方式与 <code class="xref py py-meth docutils literal notranslate"><span class="pre">FilesPipeline.item_completed()</span></code> 方法，但使用不同的字段名存储图像下载结果。</p>
<p>默认情况下， <a class="reference internal" href="#scrapy.pipelines.images.ImagesPipeline.item_completed" title="scrapy.pipelines.images.ImagesPipeline.item_completed"><code class="xref py py-meth docutils literal notranslate"><span class="pre">item_completed()</span></code></a> 方法返回项。</p>
</dd></dl>

</dd></dl>

</div>
<div class="section" id="custom-images-pipeline-example">
<span id="media-pipeline-example"></span><h2>自定义图像管道示例<a class="headerlink" href="#custom-images-pipeline-example" title="永久链接至标题">¶</a></h2>
<p>下面是图像管道的完整示例，其方法如上图所示：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>
<span class="kn">from</span> <span class="nn">itemadapter</span> <span class="kn">import</span> <span class="n">ItemAdapter</span>
<span class="kn">from</span> <span class="nn">scrapy.exceptions</span> <span class="kn">import</span> <span class="n">DropItem</span>
<span class="kn">from</span> <span class="nn">scrapy.pipelines.images</span> <span class="kn">import</span> <span class="n">ImagesPipeline</span>

<span class="k">class</span> <span class="nc">MyImagesPipeline</span><span class="p">(</span><span class="n">ImagesPipeline</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">get_media_requests</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">image_url</span> <span class="ow">in</span> <span class="n">item</span><span class="p">[</span><span class="s1">&#39;image_urls&#39;</span><span class="p">]:</span>
            <span class="k">yield</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">image_url</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">item_completed</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">results</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span>
        <span class="n">image_paths</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="s1">&#39;path&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">ok</span><span class="p">,</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">results</span> <span class="k">if</span> <span class="n">ok</span><span class="p">]</span>
        <span class="k">if</span> <span class="ow">not</span> <span class="n">image_paths</span><span class="p">:</span>
            <span class="k">raise</span> <span class="n">DropItem</span><span class="p">(</span><span class="s2">&quot;Item contains no images&quot;</span><span class="p">)</span>
        <span class="n">adapter</span> <span class="o">=</span> <span class="n">ItemAdapter</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
        <span class="n">adapter</span><span class="p">[</span><span class="s1">&#39;image_paths&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">image_paths</span>
        <span class="k">return</span> <span class="n">item</span>
</pre></div>
</div>
<p>要启用自定义媒体管道组件，必须将其类导入路径添加到 <a class="reference internal" href="settings.html#std-setting-ITEM_PIPELINES"><code class="xref std std-setting docutils literal notranslate"><span class="pre">ITEM_PIPELINES</span></code></a> 设置，如以下示例中所示：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ITEM_PIPELINES</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&#39;myproject.pipelines.MyImagesPipeline&#39;</span><span class="p">:</span> <span class="mi">300</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="deploy.html" class="btn btn-neutral float-right" title="部署蜘蛛" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="leaks.html" class="btn btn-neutral float-left" title="调试内存泄漏" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        
        &copy; 版权所有 2008–2020, Scrapy developers
      <span class="lastupdated">
        最后更新于 10月 18, 2020.
      </span>

    </p>
  </div>
    
    
    
    Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
  
 
<script type="text/javascript">
!function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","page","once","off","on"];analytics.factory=function(t){return function(){var e=Array.prototype.slice.call(arguments);e.unshift(t);analytics.push(e);return analytics}};for(var t=0;t<analytics.methods.length;t++){var e=analytics.methods[t];analytics[e]=analytics.factory(e)}analytics.load=function(t){var e=document.createElement("script");e.type="text/javascript";e.async=!0;e.src=("https:"===document.location.protocol?"https://":"http://")+"cdn.segment.com/analytics.js/v1/"+t+"/analytics.min.js";var n=document.getElementsByTagName("script")[0];n.parentNode.insertBefore(e,n)};analytics.SNIPPET_VERSION="3.1.0";
analytics.load("8UDQfnf3cyFSTsM4YANnW5sXmgZVILbA");
analytics.page();
}}();

analytics.ready(function () {
    ga('require', 'linker');
    ga('linker:autoLink', ['scrapinghub.com', 'crawlera.com']);
});
</script>


</body>
</html>